Laptop computers with integrated trackpads familiarized and popularized the concept of using a touch surface to control a graphical user interface (GUI). The operating systems which hosted the GUI on such devices treated the trackpad almost interchangeably with a mouse or similar pointing device because the fundamental point-and-click paradigm of the GUI was suitably mapped to basic trackpad operations such as touching, sliding and tapping. Over the years, a few “gestures” were added to expand the control vocabulary of the trackpad, but the underlying paradigm of manipulating an on-screen pointer remained the primary focus.
With the more recent development of touch surfaces integrated into an upper layer of a display device (such as in a tablet computer), specialized user interfaces have emerged which do not rely on the on-screen pointer paradigm, but rather utilize the direct mapping of touches to screen coordinates. There are ergonomic tradeoffs, however, in such a pointerless screen-mapped UI, so it will not replace pointer-based UIs, but the two paradigms will both continue to provide useful UIs for touch input.
Touch-based input devices (such as a trackpad) running under a pointer-based UI (such as Microsoft Windows or MacOS) present various shortcomings that limit their broader application to many tasks. Specifically, the trackpad and other touch-based input devices are often limited to point, click, drag, and scroll operations. As such, the speed with which a user can invoke specific output actions (such as launching an application, opening a web site, copying the selected text, etc.) is limited because of the visual identification, spatial cognition, and fine motor skills that are always a prerequisite to invoking a particular output action. Specifically, to invoke an output action, the user first identifies the on-screen location of the symbol (sometimes called a “user interface element”) that identifies the desired output action. The user then moves a pointer that is controlled by the touch input device from its current location to the location of the symbol, and then issues a tap to invoke the output action. The user must perform the same visual, mental and physical steps each time the user desires to invoke the same output action. Since the pointer is at an arbitrary starting location before each invocation, the pointer trajectory will be different, and the user is unable to develop and leverage muscle memory for repeated output action invocation because the actual physical motion is not consistent.
For example, when using a touch-based input device running under a pointer-based UI to enter text, a traditional “qwerty” on-screen keyboard may be displayed with clickable symbols representing the alphabetic characters. To type the word “were”, for example, the user first moves the pointer to the location of the “w” symbol and taps, then moves to the right to the location of the “e” symbol and taps, moves right again over the “r” symbol and taps, and finally moves left back to the “e” symbol and taps. Note that the “e” character was inserted two times, but the actual physical movement of invoking the first “e” involved sliding to the right and tapping, and the second “e” involved sliding to the left and tapping. This violates the first rule of muscle memory in that the same exact output action was invoked with a completely different motion. Because of this phenomenon, using a touch device to enter text on a pointer-based OS (without the enhancements provided in the present invention) is impractical and unpleasant to use. The same inefficiencies and frustration inure to other frequently used output actions in a traditional pointer-based UI. The more frequent the use, the higher the user's likely aggravation quotient.
A further limitation of touch input devices running under a pointer-based UI is that the symbol that identifies the desired output action may be hidden within folders, or hidden within a hierarchy of option screens, or hidden behind on-screen windows or within menus, etc. The user must therefore either memorize these hierarchical constructs, or search through the various menus, option screens, etc. until the symbol is finally visible and accessible on-screen. Even when the exact pathway to a desired output action is fully known because of repeated use (such as perhaps a sub-menu item within another menu item), there is a time expenditure and lack of spontaneity always associated with the precise physical movement sequence required for each invocation.
Yet a further limitation of touch input devices running under a pointer-based UI is the visual acuity and development of fine motor skills that is required in order to reliably invoke desired output actions. This is especially true for touch input, which, unlike other input modalities such as a mouse or pen, normally leaves the fingers unsupported by the palm. The degree of fine motor skills is especially apparent when output actions are only accessible within a hierarchical construct, when symbols are small or closely spaced, when the user is navigating a complex UI, and when utilizing a large screen or multiple screens.
Touch input devices running under newer screen-mapped UIs are currently used in media and gaming devices such as the iPod Touch®, smartphones (e.g., Apple's iPhone® and other smartphones powered by the Android or Windows Phone 7 operating systems), tablet computing devices such as the iPad™, automobile console controls, and a wide variety of other devices. These screen-mapped UIs allow users to utilize touch input to more directly interact with symbols, widgets, and other elements that they perceive on-screen. While this can provide significant benefit, there are still many shortcomings shared with pointer-based UIs including those outlined above, as screen-based UIs often emulate traditional keyboard and mouse functionality.
For instance, to type text on a screen-mapped touch UI, users normally insert characters by tapping on a virtual keyboard that is graphically depicted on-screen. However, when emulating a keyboard in this manner, a large amount of screen space is relegated to displaying the virtual keyboard. For smaller devices with less screen space (e.g., smartphones), the virtual keyboard occupies a majority of the screen, thereby reducing access to other functionality. If the virtual keyboard is made smaller, however, the user is more likely to make errors when using the virtual keyboard to insert characters. In an attempt to compensate for this, some manufacturers have resorted to predictive text and auto dictionary correction methods, though these methods only work in some cases and may be distracting or confusing for some users.
When emulating mouse functionality, users can invoke different output actions by tapping directly on various on-screen symbols in a manner similar to clicking or tapping on the on-screen symbols indirectly in a pointer-based UI. There are benefits and tradeoffs to this approach, and there remain inefficiencies including those described above which limit the speed and accuracy with which users can invoke output actions, including the arrested development of muscle memory for repeated output action invocation. Additionally, the hand and fingers themselves often obscure the identification of symbols, and the fingers do not provide a precise point at which object delineation is apparent.
Some pointer-based and screen-mapped UIs utilize a limited set of predefined touch gestures to improve upon the above-described shortcomings. However, these predefined touch gestures are inconsistently applied across applications, and certain gestures perform different operations in different applications or are ignored in other applications. Also, it is difficult for the user to readily identify or discover the gestures that may be available or applicable, or the techniques required to invoke them properly.
Accordingly, there is a need for enhancements to both pointer-based and screen-mapped UIs that allow users to better interact with computers and devices using existing touch hardware. Specifically, there is a need to improve how users invoke frequently used output actions when using a touch-based input device, such as launching favorite applications or web sites. There is a further need to increase the accuracy and degree of comfort with which users can invoke output actions when using touch devices so that the user does not have to guess at which output actions will be executed. There is a further need to relieve users from the need to memorize the location of hierarchically confined symbols, or search through multiple screens or other UI constructs to locate a desired symbol. There is also a need to develop and leverage the user's innate muscle memory so that with repeated use output actions can be performed with little mental, visual or physical effort. There is also a need to move users away from traditional emulated virtual keyboards so that character insertion can occur more quickly and accurately, while requiring little or no screen space.